Events listing, deletion and archival optimizations#11443
Open
winterhazel wants to merge 7 commits into
Open
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
listEventsanddeleteEventsAPIs execute some non-optimized queries, which results in very long response times in environments that have many events.Some optimizations were performed in these two workflows in order to reduce response times. The listing now uses a covering index, and does not perform a unecessary
DISTINCTanymore. The deletion is now performed in batches. The batch size is defined by the global settingdelete.query.batch.size.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
How Has This Been Tested?
Events listing
In a test environment with 2 million event entries, I called
listEventsmultiple times passing different combinations ofaccount,archived,domainid,duration,enddate,entrytime,id,keyword,level,projectid,resourceid,resourcetype,startdate,startidandtype. For almost all the listings, I got a more reasonable response time (between 0.1 and 3 seconds depending on the filters) compared to before (30+ seconds). The only exception is thekeywordparameter, which is not optimizable with a index because it performs acolumn LIKE "%keyword%"for 3 different columns.Access control
I called
listEventsand verified that:Events deletion
delete.query.batch.size, enabling batch delete;deleteEventsto remove all events before2025-01-01. Then, I verified that they were removed successfully;event.purge.delay, enabling the automatic deletion of events older than 15 days. After the deletion task executed, I verified that events older than 15 days were removed successfully.Access control
I called
deleteEventsand verified that: